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ABSTRACT 

This paper outlines pur p os es of program evaluation 
and provides practical information to help individuals from a wide 
spectrum of backgrounds use evaluation to move their program ahead in 
a purposeful manner* Wisely moving ahead requires an assessment of an 
organization's past and present as well as an examination of its 
assumptions about the future* A program evaluation can address these 
areas and inform planning* Defined as the systematic "formal" 
assessment of a program's quality and effectiveness, evaluation 
should both point out program weaknesses and highlight strengths* 
Determining and communicating the main purpose of a program review is 
essential to evaluation effectiveness* Some purposes for a program 
review are categorized under accountability, program improvement, and 
information dissemination* Various models outline the large scale 
methods of program evaluation* Smaller scale methods, or data 
gathering tools, are listed* The outcome of any review is affected, 
not only by its purpose, but also by who conducts the review and who 
the intended consumers are* Potential roadblocks and problems are 
identified and listed, and a sample timeline is provided* Terms of 
reference are data gathering guidelines that provide agreed-upon 
parameters for the review process* A sample set of terms of reference 
drawn from a school's outdoor physical education program covers 
program dimensions, planning, balance, instructional strategies, 
student evaluation, intramural activities, professional development, 
and work environment* Eight program review tips and strategies are 
provided* (SV) 
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Moving Ahead: 

Program Review & Evaluation as 
Tools for Growth 



Bruce Hendricks 
University of Calgary 



Abstract— This paper reviews the need for program evaluation. 
It discusses methods, evaluation, data gathering tools, and 
evaluation timelines. 



“The primary purpose of evaluation is not to prove but to improve” 

StufQebeam & Cuba 

“ITie educational innovator usually has to design an evaluation without much professional 
advice, conduct it with limited resources, and d^end it without the advantage of 
comparisons to conventional approaches that have been similarly evaluated. ” 

S. Hamilton 

Evaluatiiig Your Own Program 



Introduction 

Program evaluation is an often delicate and intimidating subject. It is viewed by many as something 
that someone else does to them. The goal of this article is twoMd; firstly, to provide a brief background 
regarding the various purposes and methods that evaluation can serve, and secondly, but most 
inqiortantly, to provide practical information and experience to help individuals firom a wide spectrum 
of backgrounds use evaluation as a means to move their organizations ahead in a purposeful maimer. 

Wisely moving ahead requires an assessment of an organization’s past and present as well as an 
examination of their assumptions r^arding the future. A program evaluation can help address such 
questions and consequently inform p lanning so that it is based on vdiat has been discovered, vdiat has 
been confirmed and what is anticipated. Service is vhat good evaluation is about Awell constructed 
and conducted evaluation is a service to the organizadcm, its stakeholder and its clients. Evaluation 
serves the needs of a program by providing information that is useful for making practical dioices 
regarding quality and effectiveness. 

My goal in this article is not to provide a crash course in researdi-oriented educational evaluation 
but to provide tools, insights and resources that can help you develop and conduct a program review 
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that is thorough, realistic and usable. 

What is Evaluation 

Evaluation has a long history. From time immemorial people have judged the quality and 
elSectiveness of all sorts of things and experiences. Based upon their judgment they have assigned 
value or worth. Methods of hunting, ways of treating illnesses and injuries, training for sport, 
approaches to human mteraction, art and wilderness leadership have all been hot topics of debate. 
Formal or informal methods of evaluation have often been relied onto support one position or another. 

Charting the Future; Program Review & Evaluation 
as Tools for Growth 

Today we talk of evaluating outdoor leaders as a prerequisite to giving them a stamp of approval, 
such a certification. Such behavior is hardly new. Over 4,000 years ago the Chinese evaluated key 
public officials every three years to determine their fitness for office. As individuals involved in a 
variety of outdoor programs we are concerned about the effecdvoiess of what we do both in terms 
of quahty of experience for participants and in terms of economic viability. Sometimes we need to 
justify the value of our efforts to outsiders or to others within our own organization. Evaluation is one 
way of establishing or supporting the worth of our program. Toward these ends it will be helpful to 
more specifically define evaluation. 

Evaluation as I will be addressing it in this article is a systematic, “formal" assessment of 
the quality and effectiveness of a partiatlar program. A few adHitinnal words of clarification may 
be helpful here. Because of an abundance of negative baggage associated with the term “evaluafion” 
there has been a widespread shift to the use of other terms A^ch have a similar meaning but a less 
threatening demeanor. Review and assessment are several of the evaluation synonyms that have 
gained popular usage. Many individuals perceive the purpose of evaluation as finding out “vdiat’s 
wrong” and th^ using such information, usually in the form of immutable numbers (‘‘You can’t argue 
with the data”) to embarrass and humiliate people into better performance. The idea, so the reasoning 
goes, is to motivate individuals or programs to do a better job by showing them how mudi room there 
is for inqirovement. Generally, however, this deficiency approadi to evaluation is just p lain 
demoralizing. Evaluation should not only be concerned with pointing out what could be done be^r it 
should also strive to highlight a program’s strengths. For our purposes then we will use the term 
program review interchangeably with evaluation to refer to the assessment of a program using 
specific and systematic procedures which result in findings that are useful to decision makers 
for the purpose of helping them better shape and achieve their goals. 

Purposes of Evaluation 

Sergiovanni (1987) has proposed three categories to explain wiiy evaluations should be imdartfilffai 
and what primary purposes they. Quality control 1- insuring program goals ate being achieved in a 
marmer consistent with program values and that outcomes or inputs which are nnintendfid but positive 
or important are recognized. 

Professional development - helping individuals involved in p lanning and deliverin g the program to 
grow personally and professionally by continually expanding and enhancing their own knowledge 
attitudes. Motivation of individuals involved in the program - building and nurturing motivation and 
commitment to the program and its goals. This includes the program’s ability to take care of its 
errqiloyees. 

Outdoor Programs conduct program reviews for many different reasons; field supervisors want 
to know what program elonents are effective and whidi elements can be improved on; field staff want 
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participant feedback on their effectiveness as teacher, &cilitator and outdoor leader; the board of 
directors is concerned about the thoroughness and effectiveness of safety procedures for both 
program fiicilities anH field operations; the administrator is concerned about a new program s cost 
effectiveness. Additional reasons include those vdio want to discredit a program or conversely to 
promote it; either interest group may conduct a review as a means of justifying their particular action 
orposition. 



Determining anrf communicating the main purpose of a program review is essential to ensuring 
evaluation effectiveness. Because of widely disparate and sometimes conflicting intentions within an 
organization it is inqierative for all stakeholders concerned to be clear r^arding the actual purpose of 
the program review process. What do you want to accon^ilish or find out? How will the program 
review results help you in this regard? What form will the finished report take and vibo will see it? What 
fa ulting action will be taken and vv^o will be responsible for implementation and follow up? Poor 
Hfffinitinn of purpose is a recipe for misunderstanding, resentment, defensiveness and general disregard 
for the finHingg . Ideally the results of a review will inform and influence decision makers, but in reahty, 
to cite only one exairqrle, if the findings are not in keeping with the options of those who requested or 
required the review, the results may be conveniently misplaced for a few millennia and the desired 
course of action taken irrespective of the review results. Some of the purposes for a program review 
are categorized below. 



accountability to: 

• program staff 

• participants/clients 

• funding sources (government, private agencies, tajqiayers, sponsors) 

• governing bodies (government, school boards, professional groups) 

• board of directors 

• parent organizations 

program improvement through: 

• revising program goals and objectives 

• increased recognition and understanding of successful strategies already in use 

• assessment of intended or unintended outcomes (performance) 

• more efficient and economical operation 

• establishing or improving community contacts 

• identification of staff training needs and desires. 

• inqiroving logistical procedures 

• inqiroving equipmoit selection, use and maintaiance 

• updating program content information or delivery methods 

information dissemination 

• networking with peers involved in die review process 

• hi ghlighting the effectiveness of a program 
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Methods of Evaluation 

Just as there are many reasons for conducting a program review, so too there are many ways of 
conducting a review. Many of the formal and informal evaluation methods and data gathering tools 
used in outdoor fields su(^ as e^qrerimental education have their roots in the field of educational 
evaluation. There are a great many comprehensive evaluation methods that can be used to conduct 
a review. Some are quite simple virile others are unlikely to be used by any other than a trained 
educational evaluator and a specialist support team. There is a definite difference betweoi statistics 
oriented, large scale evaluations focused on natiaiwide programs and conducted by trained evaluation 
teairrs and the less formal evaluation done by a program administrator of their own program. The 
latter however is closer to reality for most outdoor programmers. The challenge is to do the best 
possible program review with the available resources. The hardest to come by resources are usually 
time, money and trained evaluators. For this reason I have chosen to focus on a model of evaluation 
that is sinqrle to use and easily adapted (see the accompanying model The Program Review Process). 
This model was developed primarily by Dan Gxrney of Alberta Education and contributed to by a 
host of others including myself. It incorporates a variety of popular and widely used data gathering 
tools. It is a process I have used personally and which I find practical with a wide spectrum of 
programs and people. 

Having stated my own preference and focus it is important to note that there are an amazing 
number of evaluation models out there and the curious amongst you are encouraged to consult the 
second edition of W. James Popham’s witty and thorough text Educational Evaluation for more 
information. 

Data Gathering Tools 

Models are the large scale methods of program evaluation. The smaller scale methods are 
usually referred to as data gathering tools. They are simply a way of collecting information that can 
be analyzed. The review process may include many data gathering tools, tools wiiich come in all 
shares and sizes. Many of those listed below will probably be femiliar to you in one form or another. 

• participant observations 

• review of documentation 

• performance tests 

• criterion referenced tests 

• questiormaires 

• interviews (recorded, videoed or written) 

• individual (participants, instructor-teachers, administrators etc.) group 

• instructor / teacher self assessments 

• journal analysis 

• site assessment 

• at4ask analysis 

• instructional strategy analysis 

• “in-flight” corrections 

Conductors and Consumers of Evaluation 

Mudi of the value of a program review has to do with who conducts the review and who requests or 
requires it. Listed below are some of the more common conductors and consumers of evaluation. 
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conductors 
program director 
director and staff team 
outside e:q)erts 
peers in the field 
professional evaluator 



consumers 

staff 

administiatiai 
board of directors 
funding sources 
government 



Evai when the purpose ofa program review has been e^licit the possibilities for hidden agendas 

are rife. No reviewer is a perfectly unbiased observer but the quahty of the reviewer is mirrored by 
the value of the evaluation results. Especially when reviews are conducted internally but their results 
circulated externally there is the possibility of reviewers not seeing or reporting those things that might 
reflect unfiivorably on the program. 



Three factors will affectthe outdoor of any review: 



1. The purpose of the program review - if the stated purpose of a program review is 
ambiguous or a blatant misrepresaitation of reality the results are likely to be 
worthless and unusable. Specific and manageable terms of referaice generated 
coUaboratively are the ideal. 

2. Who conducts the program review - it is crucial to identify reviewers who can 
a cc omplish the stated purpose in an efficient and effective manner. They may be 
drawn fixim w ithin the program itself^ from peers outside die program or from 
some agency sudi as a department of education or a professional group 
(Hamilton, 1980; Duckett, Strother & Gephart, 1982) 

3. Who will use the results and when -his important that the reviewers understand not 
only the purpose ofthe review but also who h will be used by and whai. Ifthe final 
report will be used by the board of directors to make decisions regarding capital 
expenditures thoi they must receive the final draft with enough time to review it prior 

to their annual rtieeting rather than receiving h three daj^ after die meeting is over. 
Likewise ifthe final report is to be used by paperwork swanqied administrators 

h must be formatted in a way that makes h quici and easy to read bih whidi includes 

enough detail to be useful for decision making. 

Roadblocks In Evaluation 

There are many potoitial roadblodcs to evaluation. Identifying the most imposing of diese before the 
review begins will go a long way toward alleviating problems down the road. Many times this is a 
matter of inclusion; including those individuals or stakeholders direcdy affected by the review so as 
to give them ownership and iiqjut. There is much less resistance to and fear of a review whidi 
people have had a personal hand in developing. Here is a starter list of roadblodcs previous groups 
and individuals have identifiMl as problems for them. No doubt you have a list of your own. 

1. Time - When can I do it? Fm too busy. 

2. Disruption of programming. 

3. Fear among staff about intent - hiddoi agendas. 

4. What is the payoff? What is the cost? 

5. Who will benefit from die evaluation? 

6. Others - add your own. 
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Timelines 

Time provide a structure to help insure that a program review doesn’t turn into a ten year 
project. In addition to a specific time fiume for completion make sure that the individual or group 
responsible for a particul^ task is identified. Develop the time fiume with careful reference to the 
program’s operating time fiame. For instance wfien looking at a mountain based adventure education 
program don’t sch^ule field observations for the week of courses udien staff are still getting used to 
one annthfT and trying to determine how they “fit” within the program. Once again check your plans 
with the relevant stakeholders (field staff, administrators, tochers, etc.) to make sure you haven’t 
overlooked anything major. Listed below is a sample time fiame. 

• Staff ratify terms of reference, evaluation process and time lines, .^ril IS 

• Review team established based on program iiq}ut^ril 30 

• Meeting with program members (staf^ administration, board etc.)May 30 



• Staff completes self ^praisal June IS 

• On-site visits and data gathering June IS 

• Review team meeting to develop draft report July 1 August 1 

• Draft report presorted to program August OS 

• Review of draft report and development of action August IS 

plan by program 

• Discussion of report, necessary changes made and action Sept IS 

plan included to yield final report ^rich is distributed 



♦This is a giiggpxtftH time line. Dates may change depending upon availability of teachers, 
instructors, reviewers etc. 

Terms of Reference 

The terms of reference are data gathering guidelines r^ch provide agreed upon parameters for the 
review process. They are also a statement of purpose in more specific terms. They specify udiat 
will be evaluated and by wdiat standards. This can go a long way toward making the reviewer’s job 
more manageable and as a result making the final report more relevant and useful. The terms of 
reference shown below were designed for the review of a school physical education program vriiich 
placed an enqihasis on outdoor pursuits. 

A. Program 

Are the “thrusts” of the physical education curriculum guides in place? e.g.; 

Are the various dimaisions being offered (aquatics, dance, fitness, gymnastics, individual 
activities and outdoor pursuits)? What is offered? 

Is the movement ^proach being addressed at the elementary level? How is it implemented? 
Is the levels sqiproach being used at the secondary level? How is it inqilonented? 

B. Planning 

What statements are made regarding philosophy, goals, objectives, outcomes? 

What instructional p lanning is in place? e.g. year, unit, daily documents, tea ching resources. 

C. Balance 

What is the balance of time for activities offered in each dimension? 
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D. instructional Strategies 

W hat strategies are being used? Are they effective? 

E. Student Evaluation 

What are the criteria? Are they appropriate? How are they comraunicated? How are they 
assessed? 

F. intramurals 

What activities, clubs, special days and self-directed activities are being offered? 

G. Professional Development 

What plans exist? How are PD eiqieriences coordinated between divisions? 

H. Work Environment 

What are the arrangements for &cilities, office space, showers etc.? 

Program Review Tips and Strategies 

1. Clearly state the purpose of the review. Set specific objectives. What is going to be 
reviewed and how? 

2. Keep the number of things being looked at to a m a nag ea b le level. 

3. S election anrf t raining of evaluation team members is important. Training should fixus on the 
skills of planning, communication, observation, analysis, problem-solving and conflict resolu- 
tion in additinn to knowledge about and eiqierirace with the content area. 

4. Feedback ofa general nature is useless, e.g. “lacks organization”. Be specific enough to meet 
the client’s ne^ giving observed exaiiqiles to illustrate your points. 

5. Insure that the final rqiort is presoited in a format and manner that is usefiil to the client. 

6. Timingisirrqwrtant Thisistruefi)rplanningobsavations,presenting£eedbackandscheduling 

release of the reports to cite only a few exanqiles. 

7. Tnyolypt staff in the process of selecting the terms of reference and give them a say in the steps 
of the review. 

8. Evaluators should strive to make the review process as collaborative between evaluators and 

program monbers as possible. 

Conclusion 

The program review process is a valuable and realistic means by which programs can assess specific 
aspects of their operation and thereby help chart their organizations move ahead. There are many 
purposes for lining a pn^ram review and many individuals or combinations of individuals vriio may 
serve as conductors of the review process. The conductors may come fi'om within the organization 
itself firom outside or a combination of the two. 
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There are many methods that may be used to cany out an evaluation. The Program Review 
Process has been presented as a straightforward and ad^table method wUch can be modified to meet 
a diversity of needs. The process presented is not meant to be a research method tested for reliability 
and validity. Instead it is designed to be a practical and usable tool to aid individuals and organizations 
in finding out more about the program. 

There are certainly limitations to evaluation. One reality in the world of evaluation is that of politics 
and hidden agendas. For this reason understanding and making explicit the real purpose of an 
evaluation is paramount to its effectiveness. This is not always possible and in such cases evaluation 
may be no more than a show piece that serves the interests of one particular stakeholder. Other 
evaluation limitations include lack of money, lack of time, poor timing, poor collaboration with 
stakeholders during the evaluation design, terms of reference that are too broad and ineffective 
evaluators. However, evaluation is also full of potential benefits and the Program Review Process 
is one method to help you construct and conduct an evaluation that fits your needs and resources. When 
used carefully it will greatly fiicilitate a well designed and conducted review. Good luck in your efforts . 



Bruce Hendricks is a faculty member at the University of Calgary in the Outdoor Pursuits 
Program. 
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